FEAT add AgentThreatRulesScorer (ATR taxonomy scorer) - #1893

Freshened this onto the latest main — it is up to date and mergeable now. Ready for review when convenient.

Thanks for the thorough review, Adrian — all addressed in the latest push:

• Severity selection no longer relies on pyatr's ordering: the scorer now sorts hits by severity explicitly before picking max_severity.
• Wired pyatr>=0.2.6 as an optional atr extra and into the all group so CI installs it and the live tests actually run (kept out of dev per the functional-deps convention).
• Moved test_atr_scorer_rejects_invalid_min_severity out from behind the module-level skip — it asserts the ValueError that's raised before the pyatr import, so it no longer needs the engine. The three engine tests are now gated individually with a skipif marker.
• Fixed the assertions: benign now checks == {} (the model validator coerces None), and dropped the vacuous is-not-None line.
• _build_identifier now includes rules_dir so custom rulesets don't share an eval_hash.
• ruff clean: Optional[...] -> X | None, plus the Raises/Returns/D213 docstring fixes.

Verified locally: ruff passes, and against pyatr 0.2.6 the injection string trips 5 rules (top severity critical) while the benign string trips none, so the severity-floor tests hold. Ready for another look when you have a moment.

Thanks for the thorough review — all points addressed (pushed in 6010cb0).

CI exercising the scorer (the root cause): wired pyatr>=0.2.6 into the optional extra and the all group, so CI installs it and the three live tests run instead of being skipped. Moved test_atr_scorer_rejects_invalid_min_severity out from behind the engine gate (it only asserts the ValueError raised before the pyatr import) and switched the remaining three to a per-test requires_pyatr skipif.

Test assertions: the benign path now asserts score_metadata == {} (the no-match contract) instead of is None; dropped the vacuous is not None in the injection test, leaving the ["matched_rule_ids"] check as the real one.

Severity robustness + casing: the filtered hits are now sorted by our own _SEVERITY_ORDER (critical-first) and we take the top, instead of trusting pyatr's internal ordering. max_severity (and the description) now store the lowercased value the filter/sort compares against, so it stays correct even if pyatr ever emits mixed case.

Style: ruff check passes locally now — Optional[...] → X | None, D213 summaries, and the missing Raises: / Returns: sections added. _build_identifier now includes rules_dir, matching how SubStringScorer includes substring.

The checks here are still gated (action_required) on the fork PR — would you mind approving/re-running the workflows so the now-installed pyatr tests run? Happy to tweak anything further.

Thanks Adrian. All addressed:

• ruff format: collapsed the skipif to one line and moved to an is_pyatr_installed() helper, matching is_opencv_installed() in test_video_scorer.py. ruff check and ruff format --check are both clean now.
• Install hint: the ImportError message and the class docstring both say pip install pyrit[atr] now.
• Narrowed the import guard to ModuleNotFoundError so a failure from inside pyatr isn't misreported as "not installed".
• uv.lock: regenerated with uv lock, pyatr 0.2.6 is in it.

With pyatr wired into the atr extra and the all group, the four tests run instead of skipping. I ran them locally against pyatr 0.2.6 and all four pass.

Pushed a one-line fix (a91bcb4) for the only failing check — the ty pre-commit hook on macOS. It flagged a bare dict annotation:

error[missing-type-argument]: Missing type arguments for generic class `dict`
  --> pyrit/score/true_false/agent_threat_rules_scorer.py:135

The local metadata is now annotated dict[str, str | int | float] | None, matching Score.score_metadata's own type (no new import). ruff, the unit tests, and coverage were already green — this was the sole blocker.

The new runs are sitting in action_required pending workflow approval. Could a maintainer kick off a re-run when convenient? Thanks!

Thanks for the thorough review and the approval, Adrian Gavrila (@adrian-gavrila) — the CI-exercise and style points made the PR better. Is there anything left on your side before it can merge, or is it good to go?

Roman Lutz (@romanlutz), since you'd asked about the scorer over on #1702 — this is it, approved and ready whenever you'd like to bring it in.

Adrian Gavrila (@adrian-gavrila) — heads up that I force-pushed this branch. It had drifted 102 commits behind main and was conflicting, so it could not merge despite your approval on 2026-06-16, and it has been sitting since.

Rebuilt on current main rather than merged. The conflicts were all in pyproject.toml, uv.lock and the score package exports, which churn frequently upstream — a merge commit would have made the diff harder to re-read than a clean rebuild.

The content you approved is unchanged. The diff against main is still 5 files, +245/-1, matching the original exactly:

• the scorer and its test were carried over untouched
• the export line was reapplied with a three-way merge, not rewritten
• pyproject.toml took the same five lines (the atr extra plus the entry under all)
• uv.lock was regenerated rather than merged, which is the only substantive difference — pyatr now resolves to 0.2.7 instead of 0.2.6

Tests pass locally (4 passed).

If the force-push invalidated your approval on GitHub's side, could you re-approve when you have a moment? Happy to walk through the rebuild if you would rather verify it independently first.

Adrian Gavrila (@adrian-gavrila) — rebased onto current main. Your approval from Jun 16 is still on the PR; nothing about the scorer's behaviour changed, this was purely bringing a two-month-old branch forward.

I did not just rebase and ping, because a green CI expires once the base moves. Re-verified on latest main, with the uv version and Python this repo pins in CI (uv 0.9.17, CPython 3.13):

• tests/unit/score/test_agent_threat_rules_scorer.py — 4 passed
• tests/unit/score/ in full — 1403 passed, 16 skipped
• pre-commit run over every changed file — all hooks pass, including ty and the _async suffix check
• uv lock --check passes

Three things worth flagging from the rebase, in case any of them changes what you want:

1. pyrit/score/__init__.py — the import moved because anthrax_keyword_scorer relocated to true_false/regex/ while this branch was idle. Resolved to upstream's layout; the diff on that file is now exactly the two lines this PR needs (one import, one __all__ entry).
2. The scorer's location. I left it at pyrit/score/true_false/agent_threat_rules_scorer.py rather than moving it under the new true_false/regex/ package, since that package holds single-pattern scorers and this one is corpus-driven — and because moving an approved file invites a re-review it does not need. Say the word if you would rather it sit under regex/.
3. uv.lock is noisier than the 7 pyatr lines. Adding an optional extra changes the resolution graph, so uv re-normalizes platform markers across the file. I verified this is not a local-environment artifact: uv lock on clean main with the pinned uv and Python is a no-op, and the churn appears only once the atr extra exists. Happy to drop the lock from the PR if you would rather regenerate it yourselves.

For context on why the scorer matters alongside the loader that merged in #1715: that PR gives PyRIT the ATR payloads on the attack side, but scoring the results still needs an LLM judge. This closes that half deterministically, so a run over a few thousand payloads labels reproducibly and at no token cost.

No rush from my side — flagging it only because it was merge-ready and appears to have fallen off the queue rather than been held.

Adam Lin (@eeee2345) thank you for keeping this PR up to date, I apologize for the delay here. Going ahead with the merge!